Moving Object Segmentation by Pursuing Local Spatio-Temporal Manifolds

Although it has been widely discussed in video surveillance, background subtraction is still an open problem in the context of complex scenarios, e.g., dynamic backgrounds, illumination variations, and indistinctive foreground objects. To address these challenges, we propose a simple yet effective background subtraction method that learns and maintains dynamic texture models within spatio-temporal video patches (i.e. video bricks). In our method, the scene background is decomposed into a number of regular cells, within which we extract a series of video bricks. The background modeling is solved by pursuing manifolds (i.e. learning subspaces) with video bricks at each background location (cell). By treating the series of video bricks as consecutive signals, we adopt the ARMA (Auto Regressive Moving Average) Model to characterize spatio-temporal statistics in the subspace. In the initial learning stage, each manifold can be analytically learned, given sequences of video bricks. In the real-time detection stage, we segment foreground objects by estimating the appearance and state residuals of the new video bricks within the corresponding manifolds. Afterwards, the structure of each manifold is automatically updated by the Incremental Robust PCA (IRPCA) algorithm and its state variation by estimating the state of the new brick and re-solving linear problems. In the applications, we apply the proposed method in ten complex scenes outperform other state-of-the-art approaches. Moreover, the empirical studies of parameter settings and algorithm analysis are reported as well.